Pure web data is usually accessed via APIs and returned as JSON or XML. FME supports reading that kind of data, but in this course, we focus on reading file formats stored on the web.
After completing this lesson, you’ll be able to:
The trend toward storing data in the cloud, including spatial data, is undisputed. For that reason, FME has comprehensive tools for reading web-based datasets.
Pure web data is usually accessed via APIs and returned as JSON or XML. FME supports reading that kind of data, but in this course, we focus on reading file formats stored on the web.
The easiest way to read a web-based dataset is to paste the URL into the source Dataset parameter.
Here, the workspace author reads a MapInfo TAB dataset directly from an open data catalog stored on Amazon S3.
A folder-based dataset must be compressed to a single zip file for FME to read it from the web like this; the above perfectly illustrates that requirement. Datasets can be read from non-zipped datasets, but only when they consist of a single file (such as an AutoCAD DWG file).
The URL entered into a source dataset field may refer to any resource on a web-based file storage system. For example, here, a user is reading the same MapInfo TAB dataset directly from a Dropbox link:
The key is forcing the web service to provide a direct link to the file rather than its web interface. For example, to force Dropbox to provide a direct link, you should set the dl query parameter to 1 in the URL, as in the above screenshot, instead of the default value of 0 (zero).
If the website needs authentication before providing the data, most readers have parameters to enter such information:
However, there are better ways to read data from a web service.
Besides being able to read from a URL, FME can directly access certain web services to read data. Direct access to a web service is achieved with functionality inside FME called web connections.
Web connections are created by selecting Tools > FME Options > Web Connections on the FME Workbench menu bar and clicking the plus button in the Web Connections dialog:
New web service support is added regularly to FME. With the new FME packages system, you can update web connections without updating all of FME. You can also define custom web connections and share them with other users on the FME Hub.
FME supports connections for a large number of web services - and the Manage Services button allows you to integrate any other web service you require - but some key ones are:
These are critical types because they are capable of storing data in a way that can be accessed directly from the Add Reader dialogs:
An author adds a MapInfo dataset from Amazon S3 in the above screenshot. This action opens a dialog in which they can browse Amazon S3 for the required file:
The reader is then added to the workspace and functions like any other.
Note that if you wish to use a FeatureReader transformer instead of a reader, then the same "Select File From Web" option is available.
Usually, you must authenticate when adding a new web service connection into FME. Suppose the workspace is published to FME Flow or copied to another FME Form installation. In that case, you must re-authenticate the new location. You can also make database and web connections public.
To add a new web file system to the Add Reader or FeatureReader dialogs, click Select File From Web > FME Hub Web File Systems... to browse and download packages from the FME Hub.
Besides being able to direct a reader to a web service, various transformers exist that can read files. These are similar to the list supported by readers:
| AzureBlobStorageConnector | DropboxConnector | S3Connector |
| AzureFileStorageConnector | GoogleDriveConnector | SalesforceConnector |
| AutodeskDocsConnector | HDFSConnector | SlackConnector |
| BoxConnector | OneDriveConnector | |
| CesiumIonConnector | ProjectWiseWSGConnector |
These transformers don't read data in the traditional FME sense. Instead, they copy a selected file and add its contents to an attribute or download it to the local filesystem.
Here, for example, an author is using a Creator transformer to trigger the reading of a file from Google Drive:
FME adds the file's contents to an attribute you can process as required. For example, it might be a snippet of XML that can be decoded with an XML transformer.
Connector transformers can also upload a file, list the contents of a web service, and delete files from that service.
Given the "Select File From Web" tool on readers, Connector transformers are not generally intended for reading source data. However, they could be used to download a file that is read using a FeatureReader.
Another use would be to retrieve a list of files that are then read directly using a FeatureReader.
However, it is more likely to be used to transfer written data to the web. Writers don't have an equivalent "Select File From Web" option. The best alternative is to write data with a FeatureWriter transformer and then use a Connector transformer to transfer that data to a web service of choice.